Welcome to Cryostat! 👋

Before contributing, make sure you have:

• Read the contributing guidelines
• Linked a relevant issue which this PR resolves
• Linked any other relevant issues, PR's, or documentation, if any
• Resolved all conflicts, if any
• Rebased your branch PR on top of the latest upstream main branch
• Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
• Signed all commits using a GPG signature

To recreate commits with GPG signature git fetch upstream && git rebase --force --gpg-sign upstream/main

Fixes: #731

Description of the change:

Adds logic for automatic retry when an automated rule is activated but fails. This can happen because the discovered target application is not yet actually ready to accept requests, or because of some network hiccup, etc. Previously if an automated rule activation failed then Cryostat would never retry and the rule would not take effect on the target, until some other event caused a reactivation (rule disable/reenable, stored credentials added, target modified, target lost/found).

Motivation for the change:

Increases reliability of automated rules.

How to manually test:

1. Deploy on k8s, ex. using Helm chart
2. Deploy sample application. I used the Operator's make sample_app
3. Create an Automated Rule that matches the sample application(s)
4. Use various oc rollout restart deployment quarkus-test, oc scale deployment quarkus-test --replicas=n commands to cause targets to appear and disappear. Verify that automated rules consistently activate against these targets. There may be a slight delay in activation at times due to the retry/backoff behaviour.